That's probably OK; Java users can already pretty easily get this anyway but not Python users.

Jenkins test this please

Test build #65057 has finished for PR 15000 at commit 9c57eb7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

LGTM

My only hesitation about this is that this property really only exists to print it in the shell. Is there a good use case for it otherwise? I know it's minor but want to make sure we're not just doing this for parity.

Well, here's the use case I want it for: I'm building some plugins for JupyterHub to make it more Spark-aware, and I want to be able to link the user out to the right WebUI for their kernel. Short of somehow making the launcher override the user's own SPARK_CONF_DIR to set the port manually to one that I'm already sure is open, there's no other way to do that. But I do have access to the SparkContext so with this property, I can create the link effortlessly.

Is this Java or Pyspark? In Java you can still get this property directly from the underlying SparkContext.

PySpark. I don't think anyone runs Java through Jupyter, haha.

Ah right dumb question. Yeah I think it makes some sense ... maybe not even for Java because there are lots of methods we don't plumb through because you can easily access them directly rom scala. Python, OK.

@srowen: Just to make sure I understand, are you asking me to remove the Java accessor here, and just plumb straight through to the Scala object from PySpark? Or is it fine as-is?

Looking at context.py, it seems that it accesses things directly from SparkContext via _jsc.sc() where possible. I think that means you can just expose this in Pyspark without exposing it separately in JavaSparkContext.

@srowen _jsc.sc() is JavaSparkContext, I think that's why @apetresc expose it in JavaSparkContext first

_jsc is a JavaSparkContext and .sc() is a SparkContext. That's accessible, but maybe you'll tell me that the Py4J wrapper won't play nice with Scala Option? not sure. See how other similar fields are handled I guess.

yeah, I just try the following statement and it works. But I think it is no harm to expose it in JavaSparkContext as well.

sc._jsc.sc().uiWebUrl().get()

I don't see value in exposing it, and many other things aren't exposed via JSC. It's really only things that need a different API to make sense in Java.

As requested, removed the Java property and left only the PySpark one.

I'll admit I didn't appreciate that you could access the Scala SparkContext straight from PySpark originally (I figured the property would have to be propagated through the Java wrapper first), so the urgency of this patch is much lessened for me now that I know I can just do sc._jsc.sc().uiWebUrl().get() manually anyway - but it would still be nice to have a simple property for this instead :)

It's still probably reasonable to plumb it through but I'll leave it open a bit for comments.

@davies do you have an opinion?

Test build #3280 has finished for PR 15000 at commit cb13f43.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

merged to master

Hey guys, How can I do same thing using sparkR?